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Question 2 


Intent of Question 


The primary goals of this question were to assess a student’s ability to (1) identify a potential source of nonresponse 
bias and recognize a possible consequence for interpreting the results of a survey; (2) recognize that increasing 
sample size does not remove bias; and (3) recommend an appropriate course of action to solve a practical problem 
with the use of a survey. 


Solution 
Part (a): 


Responses were received from only 98 of the 500 (or 19.6 percent) of the randomly selected families. In other 
words, 80.4 percent of the randomly selected families did not respond to the survey. To obtain a nearly 
unbiased estimate of the proportion of families with at least one child in elementary school in this school 
district who support year-round school, we would need to assume that the families that did not respond would 
have a similar level of support for year-round school as those who did respond. This would not be the case, 
for example, if families who support year-round school were more likely to respond than families who do not 
support year-round school. In such a case, the estimate of the proportion of families who support year-round 
school calculated from the responses would tend to be higher than the population proportion of families who 
favor year-round school. 


Part (b): 


No, the nonresponse bias still exists. Combining the results from the original sample with a new random 
sample of 500 families will not solve the problem. Regardless of what happens in the second sample, the 
problem of nonresponse bias will still exist in the combined sample because there would be at least 402 
nonresponses included from the original sample. 


Part (c): 


Contact the 402 families from whom responses were not received and ask their opinion on the proposal. This 
may require additional mailings or telephone calls, but it will provide better information about support for 
year-round school among all families in the school district with at least one child in elementary school. 

OR 
Take a new random sample or take a census and use an alternative strategy, such as telephone calls or in- 
person interviews, to help increase the response rate. 


Scoring 
Parts (a), (b), and (c) are each scored as essentially correct (E), partially correct (P), or incorrect (I). 
Part (a) is scored as follows: 
Essentially correct (E) if the student clearly links the effect of nonresponse to biased estimation by explaining 


why population support for year-round school would be overestimated (or underestimated) from the sample 
results. 
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Question 2 (continued) 

Partially correct (P) if the student describes a reasonable consequence in the context of the study OR describes 
a difference related to the issue between families who are likely to respond and families who are not likely to 
respond. 
Incorrect (I) otherwise. 

Part (b) is scored as follows: 
Essentially correct (E) if the student says no, taking another sample will not solve the problem with 
nonresponse bias AND explains that nonresponse bias in the original sample would result in nonresponse bias 
in the combined sample regardless of the results from the second sample. 
Partially correct (P) if the student says no, taking another sample will not solve the problem with nonresponse 
bias, but provides a weak explanation OR says that the second sample will produce similar results to the first 
sample. 
Incorrect (1) if the student says yes, combining results from the original sample and a new random sample will 
solve the problem OR says no but provides no explanation or an incorrect argument OR does not respond to 
the question. 

Part (c) is scored as follows: 
Essentially correct (E) if the student provides an explicit description of a reasonable strategy for reducing 
nonresponse in a new survey or census or by following up with families who did not respond to the original 


study. 


Partially correct (P) if a student suggests that nonresponse should be reduced or response should be 
mandatory without providing an explicit description of a reasonable strategy. 


Incorrect (1) if the student does not provide a strategy to increase response rates or suggests a strategy that 
would result in other biases. 


4 Complete Response 
All three parts essentially correct 
3 Substantial Response 
Two parts essentially correct and one part partially correct 
2 Developing Response 
Two parts essentially correct and no parts partially correct 
= One part essentially correct and one or two parts partially correct 


OR 
Three parts partially correct 
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Question 2 (continued) 
Minimal Response 
One part essentially correct and no parts partially correct 


OR 
No parts essentially correct and two parts partially correct 
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2. A local school board plans to conduct a survey of parents’ opinions about year-round schooling in elementary 
schools. The school board obtains a list of all families in the district with at least one child in an elementary 
school and sends the survey to a random sample of 500 of the families. The survey question is provided below. 


A proposal has been submitted that would require students in elementary schools to attend school on a year- 
round basis. Do you support this proposal? (Yes or No) 


The school board received responses from 98 of the families, with 76 of the responses indicating support for 
year-round schools. Based on this outcome, the local school board concludes that most of the families with at 
least one child in elementary school prefer year-round schooling. 


(a) What is a possible consequence of nonresponse bias for interpreting the results of this survey? 
Tt is polabe that the pavers who ave excited ala ty Lropasal 
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(b) Someone advised the local school board to take an additional random sample of 500 families and to use 


the combined results to make their decision. Would this be a suitable solution to the issue raised in part (a) ? 
Explain. 


No Th fo Cet <owmple had poblems bexeuse of nau rAbouse bas, 
tun at iad Sil eid i thet, samples even if 4 iS cow lived 
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(c) Suggest a different follow-up step from the one suggested in part (b) that the local school board could take to 
address the issue raised in part (a). 
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‘GO ON TO THE NEXT PAGE. 
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2. A local school board plans to conduct a survey of parents’ opinions about year-round schooling in elementary 
schools. The school board obtains a list of all families in the district with at least one child in an elementary 
school and sends the survey to a random sample of 500 of the families. The survey question is provided below. 


A proposal has been submitted that would require students in elementary schools to attend school on a year- 
round basis. Do you support this proposal? (Yes or No) 


The school board received responses from 98 of the families, with 76 of the responses indicating support for 
year-round schools. Based on this outcome, the local school board concludes that most of the families with at 
least one child in elementary school prefer year-round schooling. 


(a) What is a possible consequence of nonresponse bias for interpreting the results of this survey? 
The Surver will corclude thet post Lam fey with af least one chet 
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(b) Someone advised the local schoo! board to take an additional random sample of 500 families and to use 
the combined results to make their decision. Would this be a suitable solution to the issue raised in part (a) ? 
Explain. 


Combining, Letults when one get sf reat.lt 43 based is nck a goed 
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(c) Suggest a different follow-up step from the one suggested in part (b) that the local school board could take to 
address the issue raised in part (a). 
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2. A local school board plans to conduct a survey of parents’ opinions about year-round schooling in elementary 
schools. The school board obtains a list of all families in the district with at least one child in an elementary 
school and sends the survey to a random sample of 500 of the families. The survey question is provided below. 


A proposal has been submitted that would require students in elementary schools to attend school on a year- 
round basis. Do you support this proposal? (Yes or No) 


The school board received responses from 98 of the families, with 76 of the responses indicating support for 
year-round schools. Based on this outcome, the local school board concludes that most of the families with at 
least one child in elementary school prefer year-round schooling. 


(a) What is a possible consequence of nonresponse bias for interpreting the results of Nie survey? 
Gecavse gees oF thea SanPtle did nov 
respond , conclusion cannor be m c a 

Jynathen oe por the Families wl at heat 


aboot ula eLev ear -vouw* 
, Vem nar We p y 
at abe Can & oe aes et on pal * 
kaart) oye prefev year found Schooling so 4 1 
™ ae “of pee duak ie SHE HCE Families hid no 
COW eee ane 


(b) Someone advised the local school board to take an additional random sample of 500 families and to use 
the combined results to make their decision. Would this be a suitable solution to the issue raised in part (a) ? 


Explain. ce . 
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(c) Suggest a different follow-up step from the one suggested in part (b) that the local school board could take to 


address the issue raised in part (a). 
“Kandomly Select SOD families and 30 to Har 
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GO ON TO THE NEXT PAGE. 
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Question 2 


Overview 


The primary goals of this question were to assess a student’s ability to (1) identify a potential source of nonresponse 
bias and recognize a possible consequence for interpreting the results of a survey; (2) recognize that increasing 
sample size does not remove bias; and (3) recommend an appropriate course of action to solve a practical problem 
with the use of a survey. 


Sample: 2A 
Score: 4 


The response to part (a) identifies a possible source of nonresponse bias by suggesting that parents who favor the 
year-round school proposal may be more likely to respond than parents who do not favor the proposal. The 
consequence is that the sample proportion would tend to be larger than the proportion of the population who favor 
the proposal. Part (a) was scored as essentially correct. The first part of the response to part (b) argues that 
nonresponse bias in the first sample would persist in the combined sample. Because this argument does not depend 
on any particular type of outcome for the second sample, it indicates that the potential for nonresponse bias in the 
combined sample arises from the low response rate in the first sample regardless of what occurs in the second 
sample. This is followed by a comment on the possibility of nonresponse bias in the new sample, but the response to 
part (b) was considered essentially correct without this comment. The response to part (c) provides an explicit 
strategy to obtain more responses from those who did not respond to the survey. Calling those who did not initially 
respond is a reasonable way to increase the response rate. Part (c) was scored as essentially correct. The answer as a 
whole, inclusive of all three parts, was considered a complete response and earned 4 points. 


Sample: 2B 
Score: 3 


The response to part (a) indicates that families with stronger opinions may be more likely to respond to the 
survey, but it does not explain how this could result in a specific consequence such as the tendency of sample 
results to overestimate the proportion of the population in favor of year-round school. Part (a) was scored as 
partially correct. The response to part (b) clearly states that the nonresponse bias in the first sample will carry over 
into the results from a combined sample, even if the second sample has no nonresponse bias. Part (b) was scored 
as essentially correct. The response to part (c) suggests that the school board should follow up with those who 
failed to respond to the original survey and offer a small reward for participating. This is a reasonable way to 
increase the level of response to the original sample. (Note that simply suggesting that the school board should 
wait until more people respond to the original survey is not an example of an effective follow-up for reducing 
nonresponse.) Part (c) was scored as essentially correct. The entire answer was deemed a substantial response and 
was awarded 3 points. 
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Question 2 (continued) 


Sample: 2C 
Score: 2 


The response to part (a) identifies a worse-case scenario in which the 402 families who did not respond to the 
survey were all opposed to the year-round school proposal, but this source of bias is not clearly linked to a 
consequence of overestimating the level of support for the proposal in the population. Part (a) was scored as 
partially correct. The response to part (b) explains that taking a second random sample will not solve the problem 
of nonresponse bias because the same level of nonresponse is likely to occur in the second sample. This argument 
relies on a specific feature of the second sample—a similar level of nonresponse—and fails to make the more 
general point that the low level of response in the first sample will create a nonresponse bias problem in the 
combined sample regardless of the results of the second sample. Part (b) was scored as partially correct. The 
response to part (c) suggests that a new random sample should be performed in which survey workers would visit 
homes to increase the response rate. Part (c) was scored as essentially correct. The overall answer was judged a 
developing response and received 2 points. 
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